Assessing human reaction to a stimulus using computer vision

ABSTRACT

Systems and methods for assessing human reaction to a stimulus using computer vision are described herein. A computer can compare a first facial image with a second facial image to identify a region of a subject&#39;s face where an expressional repositioning is evident. Based on the expressional repositioning, the computer can determine an emotion exhibited by the subject.

CLAIM OF PRIORITY

This patent application claims the benefit of priority, under 35 U.S.C.Section 120, to U.S. patent application Ser. No. 11/062,424, titled“COMPUTERIZED METHOD OF ASSESSING CONSUMER REACTION TO A BUSINESSSTIMULUS EMPLOYING FACIAL CODING,” filed on Feb. 20, 2005, which ishereby incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The invention relates to methods of measuring target audience reactionto business stimuli.

BACKGROUND

Facial behavior has typically been gauged by using either comprehensivefacial measurements or selective facial measurements. Comprehensivefacial measurements include open-ended observations of what is casuallyseen or direct measurements such as the Facial Action Coding System(“FACS”) originally described in Ekman, P., Friesen, W. V., FacialAction Coding System: A Technique for the Measurement of FacialMovement, Consulting Psychologists Press, Palo Alto, Calif. (1978).Although direct comprehensive measurements are usually both timeconsuming and costly, they provide a much greater opportunity todiscover unexpected facial action than does mere observation.

Selective facial measurements are used when the facial behavior is knownin advance. The researcher must be aware of and attend only to thespecific facial actions under investigation. These include selectiveobserver judgments, or direct measurements such as (i) the MaximallyDiscriminative Facial Movement Coding System (‘MAX”) originallydescribed in (i) Izard, C. E., The Maximally Discriminative FacialMovement Coding System, (Rev. ed.), Instructional Resources Center,University of Delaware, Newark, Del. (1983), (ii) the System forIdentifying Affect Expression by Holistic Judgment (“AFFEX”) originallydescribed in Izard, C., Dougherty, L., Hembree, E. A System forIdentifying Affect Expressions by Holistic Judgments (AFFEX),Instructional Resources Center, University of Delaware, Newark, Del.(1983), (iii) the Emotion Facial Action Coding System (“EMFACS”)originally described in Ekman, P., Friesen, W., Ancoli, S., Facial Signsof Emotional Experience, Journal of Personality and Social Psychology,vol. 39, No. 6, pp. 1125-1134 (1980), and (iv) Ekman, P., Friesen, W.,Measuring Facial Movement, Environmental Psychology and NonverbalBehavior, 1 (1), Fall 1976, pp. 56-75, and (v) facial electromyography(“EMG”). A major limitation inherent in measuring facial behaviorselectively is that this approach does not usually separate emotionalfacial behavior from conversational facial behaviors, and certainbehaviors characteristic of emotion are omitted. For example, only alimited number of the action units identified under the Emotion FacialAction Coding System (“EMFACS”) (i.e., AUs 5, 6, 7, 9 and 12) arereasonably available for coding during speech.

Investigators have conducted significant research into the use ofelectronic facial tracking and measurement for various purposes. U.S.Pat. No. 4,975,960, issued to Petajan, discloses an electrical systemfor obtaining an electronic image of the mouth region in order tofacilitate and augment the ability to monitor speech.

U.S. Pat. Nos. 5,774,591 and 5,802,220, issued to Black, disclose anelectronic system for tracking human head and facial features over timeto capture relative motions for purposes of ascertaining when a sequenceof images defines a dominant facial expression. The system seeks acorrespondence between predefined facial gestures and perceived facialexpressions for the purpose of determining which gestures trigger orsupport which expressions. The system does not seek to ascertain theunderlying emotional response of a participant to a stimulus.

U.S. Pat. No. 5,802,208, issued to Podilchuk et al., discloses anelectronic system for obtaining and comparing an electronic image of aface for purposes of recognition.

U.S. Pat. No. 6,088,040, issued to Oda et al., discloses an input/outputcontrol mechanism for receiving facial image information related todifferent facial expressions and normalizing the facial imageinformation for subsequent comparisons.

Facial tracking and measurement have not been applied in the field ofcompany and marketing analysis. Consumer reaction to a marketingstimulus (e.g., a television commercial) is typically assessed by awritten questionnaire or survey (commonly through the internet) or meansof exposing participants to the marketing stimulus (e.g., having thefocus group watch the commercial) and obtaining the reactions andcomments of the members immediately after exposure to the marketingstimulus by means of a written questionnaire and/or a personalinterview. Similarly, employee reaction to a company stimulus (e.g.,presentation on planned change in the company's benefit plan) istypically assessed by using a similar survey or “focus group” technique.While generally effective, such techniques suffer from severaldrawbacks, including a tendency for survey individuals or members of afocus group to express inaccurately or imprecisely their reaction to themarketing stimulus.

Current marketing theory suggests most consumer-purchasing behavior andbrand loyalty is dependent on the many facets of the consumer's sensoryexperiences and take-away impressions—not merely on rational decisions.Marketing efforts seek to match their offerings to the potentialconsumer's experiences, but lack a sophisticated technique to measurethe emotional response or buy-in reaction of their products or services.Measurement of workforce acceptance and buy-in to business practices,including strategic initiatives, also suffer from an inability tomeasure true emotional response or buy-in.

Facial coding can be effectively used to ascertain a target audience'strue sensory experiences and take-away impressions to marketingofferings and company related issues respectively, but is a slow,cumbersome and expensive method requiring the highly specialized skillsof personnel trained and experienced in facial coding.

Hence, a longstanding need exists for a quick, reliable and inexpensivescientific technique and methodology for measuring a consumer'semotional reaction to marketing stimuli and an employee's reaction tocompany stimuli.

SUMMARY OF THE INVENTION

The invention is directed to a computerized method of assessing anindividual's reaction to a business stimulus. The method involves (a)visually recording facial expressions of a human subject as the subjectis exposed to a business stimulus so as to generate a chronologicalsequence of recorded facial images; (b) automatically detecting andrecording expressional repositioning of each of a plurality of selectedfacial features by conducting a computerized comparison of the facialposition of each selected facial feature through sequential facialimages, (c) automatically coding contemporaneously detected and recordedexpressional repositionings to at least one of an action unit, one ormore emotions, or a combination of an affiliation value and a motivationvalue, and (d) reporting a tally of at least one of the coded actionunits, coded emotions or coded combination of an affiliation value and amotivation value over a given time period along with an identificationof the business stimulus to which the subject was exposed during thatgiven time period.

BRIEF DESCRIPTION OF THE DRAWINGS

The FIGURE is a view of a human face indicating the location of severalfacial features which can be conveniently utilized in performance of theinvention.

DETAILED DESCRIPTION OF THE INVENTION INCLUDING A BEST MODE Definitions

As utilized herein, including the claims, the phrase “action unit,”means contraction or other activity of a facial muscle or muscles thatcauses an observable movement of some portion of the face.

As utilized herein, including the claims, the term “affiliation,” meansto create or enhance the emotional connection to a company, companybranded offering, and/or business practices. Affiliation can begenerally understood in terms of promoting appeal, likeability andattraction to what is being offered.

As utilized herein, including the claims, the phrase “emotion,” meansany single expressional repositioning or contemporaneous combination ofexpressional repositionings correlated to a coded unit. The expressionalrepositionings which can be coded to Action Units and then translated tothe various emotions, or directly coded to the various emotions, are setforth in Table Three herein. The emotions include anger, disgust, fear,happiness, sadness, and surprise as set forth in the Facial ActionCoding System (“FACS”), and the additional division of “happiness” intothe smile-related categories of joy, skepticism (a/k/a false smile),micro-smile, true smile and social smile.

As utilized herein, including the claims, the term “coding to actionunits,” means to correlate a detected single expressional repositioningor combination of contemporaneous expressional repositionings with aknown single expressional repositioning or combination ofcontemporaneous expressional repositionings previously recognized asdenoting a specific Action Unit whereby the detected single expressionalrepositioning or combination of contemporaneous expressionalrepositionings can be categorized as indicating the occurrence of thattype of Action Unit. Types of action units utilized in the method ofthis invention, including those established by the Facial Action CodingSystem (“FACS”), are set forth below in Table One.

TABLE ONE DESCRIPTION OF FACS SINGLE ACTION UNITS AU No. FACS NAMEMUSCULAR BASIS  1 Inner Brow Raiser Frontalis, Pars Medialis  2 OuterBrow Raiser Frontalis, Pars Lateralis  4 Brow Lowerer DepressorGlabellae; Depressor Supercilli; Corrugator  5 Upper Lid Raiser LevatorPalebrae Superioris  6 Cheek Raiser Orbicularis Oculi, Pars Orbitalis  7Lid Tightener Orbicularis Oculi, Pars Palebrali  9 Nose Wrinkler LevatorLabii Superioris, Alaeque Nasi 10 Upper Lip Raiser Levator LabiiSuperioris, Caput Infraorbitalis 11 Nasolabial Furrow Zygomatic MinorDeepener 12 Lip Corner Puller Zygomatic Major 12U Unilateral ZygomaticMajor (e.g., one side of the face) 12F Bilateral Zygomatic Major (e.g.,both sides of the face) 14 Dimpler Buccinnator 15 Lip CornerTriangularis Depressor 16 Lower Lip Depressor Labii Depressor 17 ChinRaiser Mentalis 20 Lip Stretcher Risorius 22 Lip Funneler OrbicularisOris 23 Lip Tightener Orbicularis Oris 24 Lip Pressor Orbicularis Oris26 Jaw Drop Masetter; Temporal and Internal Pterygoid 27 Mouth StretchPtergoids; Digastric

As utilized herein, including the claims, the term “coding to emotionsor weighted emotional values,” means to correlate a detected singleexpressional repositioning or combination of contemporaneousexpressional repositionings with a known single expressionalrepositioning or combination of contemporaneous expressionalrepositionings previously recognized as denoting one or more specificemotions whereby the detected single expressional repositioning orcombination of contemporaneous expressional repositionings can becategorized as indicating the occurrence of those types of emotions. Theemotion(s) coded from each detected single expressional repositioning orcombination of contemporaneous expressional repositionings canoptionally be weighted as an indication of the likely strength of theemotion and/or the possibility that the expressional repositioning was a“false” indicator of that emotion. Types of emotions well suited for usein practicing this invention are set forth below in Tables Four andFive.

As utilized herein, including the claims, the term “coding to anaffiliation value and a motivation value,” means to correlate a detectedsingle expressional repositioning or combination of contemporaneousexpressional repositionings with a known single expressionalrepositioning or combination of contemporaneous expressionalrepositionings previously recognized as denoting an affiliation valueand a motivation value whereby the detected single expressionalrepositioning or combination of contemporaneous expressionalrepositionings can be categorized as indicating a level of affiliationand motivation.

As utilized herein, including the claims, the term “expose,” means tosubmit or make accessible to one or more of the five senses of sight,hearing, taste, touch and smell. Exposure can be guided (e.g., access ismanaged, conducted and directed by a researcher serving as a guide) orunguided (e.g., access is left to the personal whims and fancy of theparticipant based upon how he/she naturally chooses to experience thestimulus). The unguided mode encompasses the ability to engage inself-directed observational recordings.

As utilized herein, including the claims, the phrase “expressionalrepositioning” means moving a facial feature on the surface of the facefrom a relaxed or rest position to a different position using a facialmuscle, which movement is indicative of a facial expression.

As utilized herein, including the claims, the phrase “facial position”means location on the surface of the face relative to positionallystable facial features such as the bridge of the nose, the cheekbones,the crest of the helix on each ear, etc.

As utilized herein, including the claims, the term “interview,” means toask at least one question relating to a business stimulus to which aperson has just been exposed. An interview may be conducted in anynumber of settings including seated face-to-face, while walking, whileexperiencing a product or retail offering or setting, in-home, remotelyvia the internet or by a mobile phone camera. etc.

As utilized herein, including the claims, the phrase “businessstimulus,” means both company stimulus and marketing stimulus.

As utilized herein, including the claims, the phrase “company stimulus,”refers to an item pertaining to the governance, organization,management, structure, policies, procedures, work-place environment,etc. of a company. Exemplary company stimuli include announcements toemployees, reporting obligations of employees, reaction to strategicgoals and policies, bonus pay policy and procedure, type ofwork-stations, etc.

As utilized herein, including the claims, the phrase “marketingstimulus,” refers to an item selected and designed to (i) advertise,promote and/or enhance the sales or acceptance of a product and/orservice, (ii) advertise, promote, improve and/or enhance customerrecognition and/or the public image of a person and/or business, (iii)position a product, and/or (iv) enhance brand identity. Exemplarymarketing stimuli include, television commercials, radio advertisements,print advertisements, billboard advertisements, point-of-purchasedisplays, live inquiry such as questions regarding pricing, questionsregarding attitudes and questions regarding purchasing preferences,store-front signage, packaging, Yellow Pages advertisements, brochures,literature used in mass mailings, web sites, individual web pages, storelayouts, interior design of a store, product appearance, productoperation, presentation materials, etc.

As utilized herein, including the claims, the term “illustration,”includes graphs, charts, and data tables.

As utilized herein, including the claims, the phrase “same media,” meanswithin a single type of media, wherein types of media are separated asbetween (i) television, (ii) radio, (iii) print, (iv) internet, (v)three-dimensional architecture and interior design, (vi) fragrance,(vii) taste tests, (viii) use tests, etc.

As utilized herein, including the claims, the term “motivation,” meansthe ability to engender engagement so as to compel an increase inacceptance. Whereas affiliation is related to valence, motivationrelates to the potency, arousal level or impact which the stimulus hasupon the participant.

As utilized herein, including the claims, the phrase “same type of goodsor services,” means goods or services which would be listed under thesame topic heading in the most current US West Dex® Yellow Pagesdirectory relative to the date of inquiry as to whether the goods orservices are of the same type.

The Method

The method is a rigorous and accurate automated method of assessingtarget audience reaction to a business stimulus utilizing facial coding(e.g., FACS) and a unique scoring system for determining the extent towhich consumers are impacted by a business stimulus and whether suchimpact was favorable or unfavorable in relation to business objectives.Such a presentation of a target audience's reaction facilitates the useof such data in making business decisions.

The method enables businesses and organizations to gain a betterunderstanding of their target audiences' sensory and emotionalexperiences or take-away impressions to business stimuli by providingthem with psycho-physiological facial coding assessments of theaffiliation and motivation value of the business stimuli. The businessstimuli can be placed within a company-based comparative or evendirectly competitive context when desired. Use of the method withcompeting business stimuli can be evaluated, providing not only asummary of physiological responses but also a comparison between twosummaries, revealing subtle differences. Use of the method enhances theability of a company to tailor product and marketing efforts to existingand prospective customers, and tailor work-place enhancement efforts.

The method possesses several advantages, including: (1) the ability tocapture the subtle, otherwise undetectable ways in which our bodiesregister and reflect our thoughts and emotions, (2) the ability tocapture subconscious physiological responses to stimuli beyond whatmight be otherwise consciously known and verbally acknowledged, (3) theability to practice real-time and frame-by-frame assessment of responsesduring the initial experience of a stimulus and/or during a subsequentinterview, (4) the ability to conduct the study in an unobtrusive andnon-invasive manner, and (5) an objective based scientific system formonitoring emotional responses which is more accurate and precise thansubjective reports.

The method can be usefully and successfully employed in connection witha wide variety of different media including specifically, but notexclusively, television programming, television advertising, radioprogramming, radio advertising, movie trailers, phone messaging systems,customer service dynamics, signage, uniforms, vehicles, offeringdesigns, print ads, direct mail, product packaging and design (includingvisual and tactile), point of purchase displays, actual packaged orunpackaged products, taste and/or smell tests for products and/orservices, store environments, corporate lobbies, logos, brand names,taglines, catalogs, entire web sites, individual web pages, work-stationsize, configuration and placement, Power Point™ presentations, meetings,company announcements, company strategies, pricing, etc.

Generally, a subject is recruited for participation as a member of asample population, sometimes after responding appropriately to severalqualification questions (e.g., are they a member of the product targetmarket). The session is recorded by a video camera positioned to ensurethat high quality images of the participant's facial expressions areobtained throughout the session. The video camera can conveniently bemounted atop a computer monitor upon which the business stimuli willlikely be presented. Other options are to mount the camera on the tableor desktop in front of the participant, mounted within a store forviewing customers as they shop, hand-held, and images provided over amobile phone held or mounted by the subject. When the sessions involvesan interaction between the participant and an interviewer, or other formof guided test, the participant should be instructed to (i) look intothe camera when answering questions, (ii) avoid any extreme or radicalhead movement during the session, and (iii) keep from touching theirface during the session.

The participant is then exposed to the business stimuli. When exposureis guided, the participant will most often be exposed to the businessstimuli one at a time, preferably in a randomized order using apreprogrammed test procedure, and can be asked one or more questionsabout the business stimuli immediately after each exposure (i.e.,interviewed), in order to gauge the participant's emotional response toeach business stimulus, interest in purchasing product or servicesand/or development of brand loyalty. When exposure is guided, theindividual controlling the testing procedure (not shown) has thebusiness stimuli prepared and obscured until needed. Care is taken toensure that the procedure of presenting and removing the businessstimuli does not distract or inhibit the participant's natural responsesto the experience.

When exposure is unguided, the participant is simply permitted toexperience the business stimuli in a natural and unguided manner, withpassive camera monitoring of the participant's interaction with thestimuli.

Upon completion of the session (i.e., exposure to the entire collectionof business stimuli and all interview inquiries, ratings, and/orexercises completed) the videotape is shut off (manually orautomatically) and the participant dismissed. Preferably, the session isrepeated for about 10 to 200, preferably about 20 to 100 participants.

The videotape of pertinent portions of the session is then analyzed bysoftware having facial feature tracking cap abilities wherebyexpressional repositioning of selected facial features by theparticipant in the videotape can be automatically detected and recorded.Software possessing such capabilities is commercially available fromNeven Vision, Inc. of Santa Monica, Calif. under the designations ffT™(facial feature tracking). A preferred set of facial features suitablefor tracking by the ffT™ facial feature tracking software and allowingthe detection of FFMs suitable for coding to AUs, Emotions, EmotionalValues and an Affiliation Value and a Motivation Value (A&M Values) aredescribed in Table Two below and depicted in the FIGURE.

TABLE TWO FACIAL FEATURES FACIAL FEATURE NUMBER LOCATION DESCRIPTION FF1Outer Edge of Brow FF2 Middle of Brow FF3 Inner Edge of Brow FF4 OuterEdge of Eye FF5 Vertically Aligned Set on the Horizontal Center of theLower Eyelid Fold and the Horizontal Center of the Upper Eyelid Fold FF6Inside Edge of Eye FF7 Bridge of Nose FF8 Tip of Nose FF9 Lateral Sidesof Nose FF10 Area Below Each Nostril FF11 Outside Edges of Mouth FF12Left and Right Medial Points on Upper Lip FF13 Center of Upper Lip FF14Left and Right Medial Points on Lower Lip FF15 Outside Edge at Center ofLower Lip FF16 Inside Edge at Center of Upper Lip and Inside Edge atCenter of Lower Lip

The software conducts a computerized comparison of the facial positionof each of the selected facial features through sequential facial imagesand automatically codes contemporaneously detected and recordedexpressional repositionings to at least one of an Action Unit, one ormore Emotions and Emotional Values, or a combination of an AffiliationValue and a Motivation Value (A&M Values) in accordance with a codingand translation system such as the specific coding system set forthbelow.

Optionally, but preferably, the software notes the timed location of anydetected smile (i.e., facial expression involving upward and outwardmovement of one or both corners of mouth (nodes 11)) so that a trainedindividual can quickly locate occurrences of a smile on the videotape,manually view the smile, categorize the smile as a positive social smile(i.e., smile accompanied by laughter which translated to joy), a neutralsocial smile (i.e., smile accompanied by a positive or neutral commentwhich translates to a social smile) or a negative smile (i.e., a smileaccompanied by a negative or sarcastic comment which translates toskepticism), and input such categorization data into the computer systemfor use in tabulating coded results.

Coding

Detected and reported FFMs are automatically coded to AUs by a computerin accordance with a predetermined code, such as that set forth in TableThree.

TABLE THREE CODING OF FACIAL FEATURE MOVEMENT (FFM) TO AU ACTION FACIALFEATURE MOVEMENT UNIT NUMBER DIRECTION AU1 FF3 Up AU2 FF1 Up AU4 FF2and/or Down FF3 In AU5 FF5 Apart AU6 FF1 and/or Down FF5¹ Together AU7FF5¹ Together AU9 FF9 or Up & Out FF9 & FF12 Up/Up AU10 FF10 and/orWider FF12 UP AU11 FF10 Wider AU12U FF11 Up & Out (one) AU12F FF11 Up &Out (both) AU14 FF11 and Slight Out & Down FF16 Tighter AU15 F11 DownAU16 FF14 and Down FF15 Down AU17 FF13 and Up FF15 Up AU20 FF10 and/orDown & Out FF11 Down & Wider AU22 FF11 and Closer FF16 Wider AU23 FF11and Closer FF16 Tighter AU24 FF16 Tighter AU26 FF16 Wider AU27 FF11 andCloser Ff16 Widest ¹Movement of FF5 together codes to an AU7 unlessaccompanies by a contemporaneous AU12F or AU12U.

Detection and coding of FFMs to AUs during verbalization of responses ispreferably limited to AUs 5, 6, 7, 9 and 12.

Translating

Coded AUs are automatically translated to Emotions and Emotional Valuesby a computer in accordance with a predetermined correlation system,such as that set forth in Table Four. Since AU12F can translate for morethan one set of Emotions and Emotional Values, additional information isrequired. When accompanied by contemporaneous laughter in the absence ofa negative or sarcastic comment, the AU12F is evaluated as an AU12L andtranslates as set forth in Table Four. When accompanied by acontemporaneous positive or neutral comment the AU12F is evaluated as anAU12P and translates as set forth in Table Four. Finally, whenaccompanied by a contemporaneous negative or sarcastic comment the AU12Fis evaluated as an AU12N and translates as set forth in Table Four. Theadditional information necessary to determine whether a coded AU12Ftranslates as an AU12L, an AU12P or an AU12N can be manually inputtedinto the computer by a trained individual after viewing that segment ofthe videotape corresponding to the coded AU12F and observing whether theAU12F is accompanied by the components of laughter, a positive orneutral comment or a negative or sarcastic comment. In the total absenceof contemporaneous laughter or comment, an AU12F translates to an AU12P.

TABLE FOUR CORRELATION OF AUs TO EMOTION(S) ACTION EMOTION UNIT TYPEWEIGHTED VALUE AU1 Fear 0.25 Sadness 0.25 Anger 0.50 AU2 Fear 0.33Surprise 0.66 AU4 Anger 0.33 Fear 0.33 Sadness 0.33 AU5 Anger 0.33 Fear0.33 Surprise 0.33 AU6 Sadness 1.00 AU6+AU12 True Smile 1.50 AU7 Anger1.00 AU9 Disgust 1.50 AU10 Anger 0.50 Disgust 1.00 AU10+AU14 Anger 0.25Distrust 1.25 AU11 Sadness 1.00 AU12U Micro smile 1.00 AU12L¹ Joy 1.00AU12P² Social smile 1.00 AU12N³ Skepticism 1.00 AU14 Disgust 1.00 AU15Disgust 0.50 Sadness 0.50 AU16 Disgust 1.00 AU17 Anger 0.33 Disgust 0.33Sadness 0.33 AU20 Fear 1.00 AU22 Anger 1.00 AU23 Anger 1.00 AU24 Anger1.00 AU26 Anger 0.20 Fear 0.20 Disgust 0.20 Sadness 0.20 Surprise 0.20AU27 Fear 0.50 Surprise 0.50 ¹AU12L is AU12F with contemporaneouslaughter in the absence of a negative or sarcastic comment. ²AU12P isAU12F with a contemporaneous positive or neutral comment, or the absenceof any comments. ³AU12N is AU12F with a contemporaneous negative orsarcastic comment, even if accompanied by laughter.

Alternatively, the computer can be programmed to bypass the step ofcoding FFMs to AUs, and simply code the FFMs directly to Emotions andEmotional Values since this correlation is known.

Translated Emotions and Emotional Values may be further translated intoA&M Values by a computer in accordance with a predetermined correlationsystem, such as that set forth in Table Five.

The Emotional Values and A&M Values may optionally be increased by anintensity factor based upon the duration of the underlying FFM whichcoded for the Emotional Value and A&M Value. For example, the EmotionalValues and A&M Values may remain unchanged (i.e., multiplied by anintensity factor of 1) when the duration of the underlying FFM is short(≦ 3/30th of a second), increased by a factor of 2 when the duration ofthe underlying FFM is moderate (> 3/30th of a second but < 5/30th of asecond), or increased by a factor of 3 when the duration of theunderlying FFM is long (≧ 5/30th of a second).

In order to prevent a few “overly expressive” participants from floodingthe data pool and thereby monopolizing the final results, the EmotionalValues and A&M Values are preferably scaled for each participant so thateach participant contributes the same weighted value to the total. Oneconvenient method of achieving this goal is to determine the percentageof each Emotion Value for each participant based upon that participantstotal Emotional Value for all Emotions, and utilize the percentage foreach Emotional Value from each participant in any further evaluation ofgroup Emotional Values and/or group A&M Values.

TABLE FIVE CORRELATION OF EMOTION(S) TO A&M VALUES A&M COEFFICIENTSEMOTION AFFILIATION MOTIVATION Anger −0.51 0.59 Disgust −0.41 0.33 Fear−0.64 0.60 Joy 0.81 0.51 Sadness −0.63 −0.27 Skepticism −0.22 0.21Surprise 0.40 0.67 Micro Smile 0.55 0.07 Social Smile 0.61 0.10 TrueSmile 0.76 0.48

Alternatively, the computer can be programmed to bypass the step ofcoding FFMs to AUs and/or translating coded AUs to Emotions andEmotional Values and simply code FFMs directly to A&M Values since thiscorrelation is known.

Reporting

A number of reporting options exists for representing the data. Briefly,the Emotional Values and/or A&M Values may be reported for eachparticipant, segmented groups of the participants (e.g., men v. women orthose who had purchased the advertised product before v. those who hadnot purchased the advertised product before) or all participants. TheEmotional Values and/or A&M Values may also be reported at selected timeintervals (e.g., every second), at selected points in the session (e.g.,one report during viewing of the stimulus and another report in responseto one or more specific questions during the interview), or the entiresession. Several exemplary options of specific types of reports includespecifically, but not exclusively, (i) reporting the data separately foreach individual stimuli, (ii) reporting the data for a plurality ofstimuli in comparative form, (iii) reporting the data separately foreach member of the sample population, (iv) graphically reporting theaverage affiliation and motivation values with each score forming anaxis of the graph, (v) graphically reporting the average affiliation andmotivation values for only a selected portion of the exposure period(e.g., the first six seconds of exposure to the business stimuli, thereaction to a specific question, etc.), (vi) graphically reporting theaverage affiliation and motivation values for a plurality of distinctportions of the exposure period together on the same graph (e.g., scoresduring exposure to the business stimuli and scores during questioning),(vii) graphically reporting the affiliation and motivation values foreach participant to ascertain the distribution of the participants,(viii) graphically reporting, in a pie chart or bar graph format, thepercentage of each emotional value to a related stimulus, etc. Onereport of particular interest is a second-by-second report of EmotionalValues and/or A&M Values throughout a television commercial.

EXAMPLES Example 1

Participants A-E are each exposed to two stimuli, asked one questionimmediately after viewing each stimulus, and asked a second questionlater regarding the ability of the stimulus to induce enhanced purchaselikelihood. The faces of the participants are videotaped during theexposure and interview periods specifically related to the questions.The videotape of each participant is analyzed using ffT™ facial featuretracking software. The facial feature movements (FFM) listed in TableSix are detected and reported. The duration of each detected FFM isdetermined, categorized as short (≦ 1/30th of a second), moderate (>1/30th of a second but < 3/30th of a second) or long (≧ 3/30th of asecond). The software tags or reports the location of each detectedmovement involving FF12 on each videotape. A trained individual reviewsthe tagged location of each FF11 related portion of videotape and inputsdata into the computer indicating the observation of any laughter,positive or neutral comment or a negative or sarcastic commentaccompanying movement of FF12.

The computer codes the detected FFMs, along with any associated datainput by the trained individual, into Action Units (AU) as shown inTable Six. The coded AUs are then correlated to weighted EmotionalValues and Affiliation and Motivation Values (A&M Values) as shown inTables Seven (A) and (B) (Emotional Values) and Eight (A&M Values). Thecomputer multiplies each correlated Emotional Value—and therebyinherently multiplies the derived A&M Values—by the correspondingintensity factor to obtain enhanced values. The categorized tally ofenhanced Emotional Values and enhanced A&M Values for each stimulus isset forth in the final rows of Tables Seven (A) and (B) (EmotionalValues) and Eight (A&M Values).

The relative percentages of the tallied value of each emotion for eachstimulus, and the averaged A&M Values (averaged for number of coded AUs)for each stimulus is automatically calculated and reported, again asshown in Tables Seven (A) and (B) (Emotional Values) and Eight (A&MValues).

TABLE SIX PARTIC- INTENSITY OBSERVATION IPANT FFM CATEGORY DATA AUSTIMULUS ONE A FF3 Up Low AU1 FF2 Down Low AU4 FF9 Up & Out Moderate AU9B FF3 Up Low AU1 FF11 Up & Out Low Laughter AU12L FF16Wider Low AU22 CFF5 Together Low AU7 FF3 Up Low AU1 FF2 Down Low AU4 FF16 Wider Low AU22D FF2 Down Low AU4 FF5 Together High AU7 FF16 Wider Low AU22 FF16WiderLow AU22 FF11 Up & Out Moderate Laughter AU12L E FF5 Together Low AU7FF2 Down Low AU4 FF5 Together Moderate AU7 STIMULUS TWO A FF16Wider LowAU2 FF11 (one) Up Low AU12U & Out FF11 (both) Up Low Negative AU12N &Out Comment B FF3 Up Low AU1 FF11 (both) Up Low Sarcastic AU12N & OurComment C FF16 Wider Low AU22 FF1 Down Low AU6 D FF11 (one) Up Low AU12U& Out FF1 Up Low AU2 E FF11 (one) Up Low AU12U & Out FF16 Widest LowAU27 FF11 (both) Up Low Neutral Comment AU12P & Out

TABLE SEVEN (A) WEIGHTED EMOTIONS INTENSE Smile Smile Smile PARTY FACTORAU Anger Disgust Fear Joy Sadness Skeptic (micro) (Social) (True)Surprise STIMULUS ONE A 1 AU1 0.25 0.25 0.50 1 AU4 0.33 0.33 0.33 1.5AU9 1.50 Totals w/ Intensity 0.33 2.25 0.58 0 0.58 0 0 0 0 0.50 % of AllEmotions 7.9% 52.9% 13.7% 0 13.7% 0 0 0 0 11.8% B 1 AU1 0.25 0.25 0.50 1AU12L 1.00 1 AU22 1.00 Totals w/ Intensity 1.00 0.25 1.00 0.25 0.50 % ofAll Emotions 33.3% 0 8.3% 33.3% 8.3% 0 0 0 0 16.8% C 1 AU7 1.00 1 AU10.25 0.25 0.50 1 AU4 0.33 0.33 0.33 1 AU22 1.00 Totals w/ Intensity 2.330 0.58 0 0.58 0 0 0 0 0.50 % of All Emotions 58.4% 0 14.6% 0 14.5% 0 0 00 12.5% D 1 AU4 0.33 0.33 0.33 3 AU7 1.00 1 AU22 1.00 1 AU22 1.00 1.5AU12L 1.00 Totals w/ Intensity 5.33 0 0.33 1.50 0.33 0 0 0 0 0 % of AllEmotions 71.2% 0 4.4% 20.0% 4.4% 0 0 0 0 0 E 1 AU7 1.00 1 AU4 0.33 0.330.33 1.5 AU7 1.00 Totals w/ Intensity 2.83 0 0.33 0 0.33 0 0 0 0 0 % ofAll Emotions 81.0% 0 9.5% 0 9.5% 0 0 0 0 0 TOTAL OF PARTICIPANT 251.852.9 50.5 53.3 50.4 0 0 0 0 41.1 EMOTION % AVERAGE OF EMOTION % 50.4%10.6% 10.1% 10.7% 10.0% 0 0 0 0 8.2 FOR PARTICIPANTS

TABLE SEVEN (B) WEIGHTED EMOTIONS INTENSE Smile Smile Smile PARTY FACTORAU Anger Disgust Fear Joy Sadness Skeptic (micro) (Social) (True)Surprise STIMULUS TWO A 1 AU22 1.00 1 AU12U 1.00 1 AU12N 1.00 Totals w/Intensity 1.00 0 0 0 0 1.00 1.00 0 0 0 % of All Emotions 33.3% 0 0 0 033.3% 33.3% 0 0 0 B 1 AU1 0.25 0.25 0.50 1 AU12N 1.00 Totals w/Intensity 0 0 0.25 0 0.25 1.00 0 0 0 0.50 % of All Emotions 0 0 12.5% 012.5% 50.0% 0 0 0 25.0% C 1 AU22 1.00 1 AU6 1.00 Totals w/ Intensity1.00 0 0 0 1.00 0 0 0 0 0 % of All Emotions 50.0% 0 0 0 50.0% 0 0 0 0 0D 1 AU12U 1.00 1 AU2 0.33 0.66 Totals w/ Intensity 0 0 0.33 0 0 0 1.00 00 0.66 % of All Emotions 0 0 16.6% 0 0 0 50.2% 0 0 33.2% E 1 AU12U 1.001 AU27 0.50 0.50 1 AU12P 1.00 Totals w/ Intensity 0 0 0.50 0 0 0 1.001.00 0 0.50 % of All Emotions 0 0 16.7% 0 0 0 33.3% 33.3% 0 16.7% TOTALOF PARTICIPANT 83.3 0 45.8 0 62.5 83.3 116.8 33.3 0 74.9 EMOTION %AVERAGE OF EMOTION % 16.7% 0 9.2% 0 12.5% 16.7% 23.3% 6.6% 0 15.0% FORPARTICIPANTS

TABLE EIGHT EMOTIONS AVERAGE % FOR A&M COEFFICIENTS A&M VALUES TYPEPARTICIPANTS AFFILIATION MOTIVATION AFFILIATION MOTIVATION STIMULUS ONEAnger 50.4 −0.51 0.59 −0.26 0.30 Disgust 10.6 −0.41 0.33 −0.04 0.03 Fear10.1 −0.64 0.60 −0.06 0.06 Joy 10.7 0.81 0.51 0.08 0.05 Sadness 10.0−0.63 −0.27 −0.06 −0.03 Skepticism 0 −0.22 0.21 0 0 Surprise 8.2 0.400.67 0.03 0.05 Micro Smile 0 0.55 0.07 0 0 Social Smile 0 0.61 0.10 0 0True Smile 0 0.76 0.48 0 0 TOTALS −0.31 0.46 EMOTIONS TOTALS OF A&MCOEFFICIENTS A&M VALUES TYPE PARTICIPANT % AFFILIATION MOTIVATIONAFFILIATION MOTIVATION STIMULUS TWO Anger 16.7 −0.51 0.59 −0.09 0.10Disgust 0 −0.41 0.33 0 0 Fear 9.2 −0.64 0.60 −0.06 0.05 Joy 0 0.81 0.510 0 Sadness 12.5 −0.63 −0.27 −0.08 −0.03 Skepticism 16.7 −0.22 0.21−0.04 0.04 Surprise 15.0 0.40 0.67 0.06 0.10 Micro Smile 23.3 0.55 0.070.13 0.02 Social Smile 6.6 0.61 0.10 0.04 0.01 True Smile 0 0.76 0.48 00 TOTALS −0.04 0.29

Conclusions: Participant reaction to stimulus one shows a strong adversereaction to the stimulus. In other words, the response may becharacterized as a “poison pill” giving significant arousal (i.e.,motivation) but a decidedly adverse response based upon the negativeaffiliation score. Anger is clearly the most prevalent response and allpositive reactions combined account for less than 20% of the reaction.In contrast, the response to stimulus two is essentially one ofambivalence. The reaction is slightly more negative than positiveoverall. The motivation score shows a lack of engagement, while theaffiliation score is all but neutral. The fact that the tepidmicro-smile category was the most common response to stimulus two showsthe unlikelihood of this stimulus to make a marked difference in thebusiness climate.

The specification is provided to aid in the complete nonlimitingunderstanding of the invention. Since many variations and embodiments ofthe invention may be created and devised without departing from thespirit and scope of the invention, the scope of the invention resides inthe claims hereinafter appended.

What is claimed is:
 1. A non-transitory machine-readable mediumincluding instructions, which when executed by a machine, cause themachine to perform operations comprising: comparing a first facial imagewith a second facial image to identify a region of a face of a subjectwhere an expressional repositioning is evident; and determining anemotion exhibited by the subject by creating a weighted emotional valuefrom the identified region.
 2. The machine-readable storage medium ofclaim 1, wherein the first facial image and the second facial image arechronologically ordered, and wherein the first facial image correspondsto the face prior to exposure of the subject to an identified stimulusand the second facial image corresponds to the face following exposureof the subject to the identified stimulus.
 3. The machine-readablestorage medium of claim 1, wherein determining an emotion exhibited bythe subject includes coding the identified region to an action unit. 4.The machine-readable storage medium of claim 3, wherein the action unitsare FACS action units.
 5. The machine-readable storage medium of claim3, wherein the action units are EMFACS action units.
 6. Themachine-readable storage medium of claim 1, wherein the weightedemotional value includes a weight based on a duration of theexpressional repositioning.
 7. The machine-readable storage medium ofclaim 1, wherein the weighted emotional value includes an indication ofthe impact the stimulus had on the subject.
 8. The machine-readablestorage medium of claim 7, wherein the impact is at least one offavorable or unfavorable.
 9. A method comprising: comparing, using ahardware processor, a first facial image with a second facial image toidentify a region of a face of a subject where an expressionalrepositioning is evident; and determining, using the hardware processor,an emotion exhibited by the subject by creating a weighted emotionalvalue from the identified region.
 10. The method of claim 9, wherein thefirst facial image and the second facial image are chronologicallyordered, and wherein the first facial image corresponds to the faceprior to exposure of the subject to an identified stimulus and thesecond facial image corresponds to the face following exposure of thesubject to the identified stimulus.
 11. The method of claim 9, whereindetermining an emotion exhibited by the subject includes coding theidentified region to an action unit.
 12. The method of claim 11, whereinthe action units are FACS action units.
 13. The method of claim 11,wherein the action units are EMFACS action units.
 14. The method ofclaim 9, wherein the weighted emotional value includes a weight based ona duration of the expressional repositioning.
 15. The method of claim 9,wherein the weighted emotional value includes an indication of theimpact the stimulus had on the subject.
 16. The method of claim 15,wherein the impact is at least one of favorable or unfavorable.
 17. Adevice comprising: a hardware processor; and a memory, the memoryincluding instructions, which when executed by the hardware processor,cause the device to: compare a first facial image with a second facialimage to identify a region of a face of a subject where an expressionalrepositioning is evident; and determine an emotion exhibited by thesubject by creating a weighted emotional value from the identifiedregion.
 18. The device of claim 17, wherein the first facial image andthe second facial image are chronologically ordered, and wherein thefirst facial image corresponds to the face prior to exposure of thesubject to an identified stimulus and the second facial imagecorresponds to the face following exposure of the subject to theidentified stimulus.
 19. The device of claim 17, wherein to determine anemotion exhibited by the subject includes the memory includinginstructions to cause the device to code the identified region to anaction unit.
 20. The device of claim 17, wherein the weighted emotionalvalue including a weight based on a duration of the expressionalrepositioning, and the weighted emotional value including an indicationof the impact the stimulus had on the subject, where the impact is atleast one of favorable or unfavorable.